{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "3-imdb.ipynb",
      "version": "0.3.2",
      "views": {},
      "default_view": {},
      "provenance": [],
      "collapsed_sections": []
    }
  },
  "cells": [
    {
      "metadata": {
        "id": "ItXfxkxvosLH",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Classifying movie reviews with tf.keras, tf.data, and eager execution.\n",
        "\n",
        "In this lab you'll learn how to classify a movie review as positive or negative on the [IMDB dataset](https://www.tensorflow.org/api_docs/python/tf/keras/datasets/imdb). This tutorial was inspired by [this work](https://github.com/fchollet/deep-learning-with-python-notebooks/blob/master/3.5-classifying-movie-reviews.ipynb). Here, we'll use tf.keras and tf.data."
      ]
    },
    {
      "metadata": {
        "id": "2ew7HTbPpCJH",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "!pip install -q -U tensorflow==1.8.0\n",
        "import tensorflow as tf\n",
        "\n",
        "# Enable eager execution\n",
        "tf.enable_eager_execution()\n",
        "\n",
        "import numpy as np"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "iAsKG535pHep",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# A first look at the dataset\n",
        "\n",
        "### Step 1) Downloading the dataset\n",
        "\n",
        "The \"IMDB dataset\" is a set of around 50,000 positive or negative reviews for movies from the Internet Movie Database. Let's use the TensorFlow API to download the dataset the same way we did for MNIST. The `num_words` argument is how many of the most popular words we will keep. All other \"rare\" words will be discarded."
      ]
    },
    {
      "metadata": {
        "id": "zXXx5Oc3pOmN",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "(train_data, train_labels), (test_data, test_labels) = tf.keras.datasets.imdb.load_data(num_words=10000)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "l50X3GfjpU4r",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Step 2) Exploring the dataset \n",
        "\n",
        "The dataset comes preprocessed. Each example is an array of integers representing the words of the movie review. Each label is either a \"0\" for negative or \"1\" for positive."
      ]
    },
    {
      "metadata": {
        "id": "QtTS4kpEpjbi",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "print(train_data.shape)\n",
        "print(train_labels.shape)\n",
        "print(train_data[0])\n",
        "print(train_labels[0])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "3KmM0e64pok8",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "\n",
        "### Word indexing\n",
        "\n",
        "To prove we have limited ourselves to the 10,000 most frequent words we will iterate over all the reviews and check the maximum value. "
      ]
    },
    {
      "metadata": {
        "id": "GvgpnR2ppqHd",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "print(max([max(review) for review in train_data]))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "4wJg2FiYpuoX",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Converting the integers back to words\n",
        "\n",
        "We can get the dictionary from `tf.keras.datasets.imdb.get_word_index` which hashes the words to their corresponding integers. Let's try and convert a review from integers back into it's original text by first reversing this dictionary and then iterating over a review and converting the integers to strings."
      ]
    },
    {
      "metadata": {
        "id": "tr5s_1alpzop",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "# Dictionary that hashes words to their integer\n",
        "word_to_integer = tf.keras.datasets.imdb.get_word_index()\n",
        "\n",
        "# Print out the first ten keys in the dictionary\n",
        "print(list(word_to_integer.keys())[0:10])\n",
        "\n",
        "integer_to_word = dict([(value, key) for (key, value) in word_to_integer.items()])\n",
        "\n",
        "# Demonstrate how to find the word from an integer\n",
        "print(integer_to_word[1])\n",
        "print(integer_to_word[2])\n",
        "\n",
        "# We need to subtract 3 from the indices because 0 is \"padding\", 1 is \"start of sequence\", and 2 is \"unknown\"\n",
        "decoded_review = ' '.join([integer_to_word.get(i - 3, 'UNK') for i in train_data[0]])\n",
        "print(decoded_review)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "lFP_XKVRp4_S",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Step 3) Format the data\n",
        "Unfortunately, we cannot just feed unformatted arrays into our neural network. We need to standardize the input. Here, we are going to multi-hot-encode our review which is an array of integers into a 10,000 dimensional vector. We will place 1's in the indices of word-integers that occur in the review, and 0's everywhere else.\n"
      ]
    },
    {
      "metadata": {
        "id": "NG4K-OtdqC0a",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "def vectorize_sequences(sequences, dimension=10000):\n",
        "    # Create an all-zero matrix of shape (len(sequences), dimension)\n",
        "    results = np.zeros((len(sequences), dimension), dtype=np.float32)\n",
        "    for i, sequence in enumerate(sequences):\n",
        "        results[i, sequence] = 1.  # set specific indices of results[i] to 1s\n",
        "    return results\n",
        "\n",
        "train_data = vectorize_sequences(train_data)\n",
        "test_data = vectorize_sequences(test_data)\n",
        "\n",
        "print(train_data.shape) # length is same as before\n",
        "print(train_data[0]) # now, multi-hot encoded\n",
        "\n",
        "# Vectorize the labels as well and reshape from (N, ) to (N, 1)\n",
        "train_labels = np.reshape(np.asarray(train_labels, dtype=np.float32), (len(train_data), 1))\n",
        "test_labels = np.reshape(np.asarray(test_labels, dtype=np.float32), (len(test_data), 1))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "jeYlLZlrqYjq",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Step 4) Create the model\n",
        "\n",
        "Now that we have vectorized input, we're ready to build our neural network. We're going to use multiple hidden layers, each with 16 hidden units. The more layers and hidden units you have, the more complicated patterns the network can recognize and learn. \n",
        "\n",
        "* The downside is that a large number potentially leads to [overfitting](https://developers.google.com/machine-learning/crash-course/generalization/peril-of-overfitting) by finding patterns that may only exist in your training not (and not generalize to unseen testing data down the road).\n",
        "\n",
        "Instead of softmax as in the previous notebook, here our last activation will be a sigmoid. This will take the output of the first two layers and \"squish\" it into a number between 0 and 1 (the network's prediction for the review)."
      ]
    },
    {
      "metadata": {
        "id": "uVOGq_eirutM",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "# Create a model\n",
        "model = tf.keras.Sequential()\n",
        "\n",
        " # input shape here is the length of our movie review vector\n",
        "model.add(tf.keras.layers.Dense(16, activation=tf.nn.relu, input_shape=(10000,)))\n",
        "model.add(tf.keras.layers.Dense(16, activation=tf.nn.relu))\n",
        "model.add(tf.keras.layers.Dense(1, activation=tf.nn.sigmoid))\n",
        "\n",
        "optimizer = tf.train.RMSPropOptimizer(learning_rate=0.001)\n",
        "\n",
        "model.compile(loss='binary_crossentropy',\n",
        "              optimizer=optimizer,\n",
        "              metrics=['binary_accuracy'])\n",
        "\n",
        "model.summary()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "nIBgYoE1sCmw",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Step 5) Create a validation set\n",
        "\n",
        "We want to test our model on data it hasn't seen before, but before we get our final test accuracy. We'll call this dataset our validation set. In practice, it's used to tune parameters like your learning rate, or the number of layers / units in your model."
      ]
    },
    {
      "metadata": {
        "id": "CfoGOjesswbb",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "VAL_SIZE = 10000\n",
        "\n",
        "val_data = train_data[:VAL_SIZE]\n",
        "partial_train_data = train_data[VAL_SIZE:]\n",
        "\n",
        "\n",
        "val_labels = train_labels[:VAL_SIZE]\n",
        "partial_train_labels = train_labels[VAL_SIZE:]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "GBvKRJIts5U4",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Step 6) Create a tf.data Dataset\n",
        "As before, we're working with a small in-memory dataset, so converting to tf.data isn't essential, just good practice for down the road. See the previous notebook or the `tf.data.Dataset` [API doc](https://www.tensorflow.org/api_docs/python/tf/data) for more info."
      ]
    },
    {
      "metadata": {
        "id": "dz9EDovvtGa6",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "BATCH_SIZE = 512\n",
        "SHUFFLE_SIZE = 1000\n",
        "\n",
        "training_set = tf.data.Dataset.from_tensor_slices((partial_train_data, partial_train_labels))\n",
        "training_set = training_set.shuffle(SHUFFLE_SIZE).batch(BATCH_SIZE)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "nh9l8Cdfte8g",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Step 7) Training your model\n",
        "Now we are going to train the model. As we go, we'll keep track of the training loss, training accuracy, validation loss, and validation accuracy. \n",
        "\n",
        "Please be patient as this step may take a while."
      ]
    },
    {
      "metadata": {
        "id": "_UVH5tQqtcxc",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "EPOCHS = 10\n",
        "\n",
        "# Store list of metric values for plotting later\n",
        "tr_loss_list = []\n",
        "tr_accuracy_list = []\n",
        "val_loss_list = []\n",
        "val_accuracy_list = []\n",
        "\n",
        "for epoch in range(EPOCHS):\n",
        "  for reviews, labels in training_set:\n",
        "    # Calculate training loss and accuracy\n",
        "    tr_loss, tr_accuracy = model.train_on_batch(reviews, labels)\n",
        "  \n",
        "  # Calculate validation loss and accuracy\n",
        "  val_loss, val_accuracy = model.evaluate(val_data, val_labels)\n",
        "\n",
        "  # Add to the lists\n",
        "  tr_loss_list.append(tr_loss)\n",
        "  tr_accuracy_list.append(tr_accuracy)\n",
        "  val_loss_list.append(val_loss)\n",
        "  val_accuracy_list.append(val_accuracy)\n",
        "  \n",
        "  print(('Epoch #%d\\t Training Loss: %.2f\\tTraining Accuracy: %.2f\\t'\n",
        "         'Validation Loss: %.2f\\tValidation Accuracy: %.2f')\n",
        "         % (epoch + 1, tr_loss, tr_accuracy,\n",
        "         val_loss, val_accuracy))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "xNzTBpLBvPPn",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Step 8) Plotting loss and accuracy\n",
        "We are going to use `matplotlib` to plot our training and validation metrics."
      ]
    },
    {
      "metadata": {
        "id": "kcBHF0rKt4Rn",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "epochs = range(1, EPOCHS + 1)\n",
        "\n",
        "# \"bo\" specifies \"blue dot\"\n",
        "plt.plot(epochs, tr_loss_list, 'bo', label='Training loss')\n",
        "# b specifies a \"solid blue line\"\n",
        "plt.plot(epochs, val_loss_list, 'b', label='Validation loss')\n",
        "plt.title('Training and validation loss')\n",
        "plt.xlabel('Epochs')\n",
        "plt.ylabel('Loss')\n",
        "plt.legend()\n",
        "\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "sx1Vg_1qvT5D",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "plt.clf()   # Clear plot\n",
        "\n",
        "plt.plot(epochs, tr_accuracy_list, 'bo', label='Training accuracy')\n",
        "plt.plot(epochs, val_accuracy_list, 'b', label='Validation accuracy')\n",
        "plt.title('Training and validation accuracy')\n",
        "plt.xlabel('Epochs')\n",
        "plt.ylabel('Accuracy')\n",
        "plt.legend()\n",
        "\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "rX_qklYwvak2",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Step 9) Testing your model\n",
        "Now that we have successfully trained our model and our training accuracy has jumped over 95%, we need to test it on an entirely different dataset. We are going to run our model on the test set. The test accuracy is a better evaluation metric for how our model will perform in the real world. "
      ]
    },
    {
      "metadata": {
        "id": "Oghmxes4vhs-",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "loss, accuracy = model.evaluate(test_data, test_labels)\n",
        "print('Test accuracy: %.2f' % (accuracy))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "bl7gZC_lvxyD",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Congratulations\n",
        "You have successfully trained a model on the IMDB dataset."
      ]
    }
  ]
}
